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Abstract 



I. Introduction 



This paper is concerned with certain connections between the ensemble of n x n uni- 
tary matrices — specifically the characteristic function of the random variable tr([/) — and 
combinatorics — specifically Ulam's problem concerning the distribution of the length of the 
^ ■ longest increasing subsequence in permutation groups — and the appearance of Painleve func- 
! tions in the answers to apparently unrelated questions. Among the results is a representation 
^ I in terms of a Painleve V function for the characteristic function of tr(f/) and (using recent 

results of Baik, Deift and Johansson) an expression in terms of a Painleve 11 function for the 
limiting distributiuon of the length of the longest increasing subsequence in the hyperocta- 
OO ■ hedral groups. 

S 

I The characteristic function of the random variable tr U, where U belongs to the ensemble 

■ U{n) of n X n unitary matrices with Haar measure, is the expected value 

^ ■ E4e'-t"^+^*^^). (1.1) 

In U{n) we have for any function g with Fourier coefficients Qk 

En (^f[9{e''^?j =detT„(^7), (1.2) 

where Tn{g) is the associated n x n Toeplitz matrix defined by 

Tnig) = i9j~k), (j, /c = 0, ■ ■ ■ , n - 1). 

It follows that the distribution function (|1 . 1| ) equals the determinant of the Toeplitz matrix 
associated with the function e'^^'^^^ . The determinant, which we denote by Dn, is a function 



1 



of the product rs (see Sec. II) and so it is completely determined by its values when r = s = t. 
This function Dn{t) has connections with both integrable systems and combinatorial theory. 
To state our results, and these connections, we introduce some notation. 
We set 



so that Dnit) = det Tn{f). Notice that T„(/) is symmetric since f{z ^) = f{z). We 
introduce the n-vectors 



5+ 



/ 1 \ 






/ \ 






( h \ 

f2 

fn-1 
\ fn J 



f- 



( fn \ 

fn-1 

f2 

\ /l J 



and define 
If we set 



then $„ function of t satisfies the equation 



1 



2V$ -1 $ 



;i.3) 



which is a variant of the Painleve V equation,[] and in terms of it Dn has the representation 



Dn{t) = exp (4 log(t/r) r $„(r) dr 



;i-4) 



This is reminiscent of the many representations now in the literature for Fredholm determi- 
nants in terms of Painleve functions. We shall also show that 



= UJU, 



n-1 



satisfies 



W" = —(W'f --W' +4: - — - W^- — + 4:W^- — 



;i.5) 



which is a special case of the Painleve III equation. 

An important ingredient in the proofs is the following recurrence relation satisfied by the 



n 



0. 



;i.6) 



We shall see that this recurrence formula, sometimes known as the discrete Painleve II 
equation (see, e.g. is equivalent to one first shown to hold for U{n) by V. Periwal and 

-'^The substitution = z and 4>„ = w/(w — 1) transforms ( pT^ ) to the standard form of Py with parameters 
a = 0, /? = 7 = 2 and (5 = 0. 
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D. Shevitz [Q. It was rediscovered by M. Hisakado who also derived an equation equiv- 
alent to (|1.3|) and observed that this was one of the Painleve V equations which, by results 
of K. Okamoto 0, is reducible to Painleve III. Carrying through the Okamoto procedure 
is what led to our Wn, although the proof we give is direct. Our derivations of (|l]3|) and 
( p..6[ ) are different from those in Q and perhaps more down-to-earth since we use only the 
simplest properties of Toeplitz matrices and some linear algebra. (They cannot be entirely 
unrelated, though, since the orthogonal polynomials which are central to the argument of 
can be defined in terms of Toeplitz matrices.) 

A remarkable connection between U{n) and combinatorics was discovered by Gessel 0. 
Place the uniform measure on the symmetric group Sjy , denote by ^Ar(o") the length of the 
longest increasing subsequence in cr, and define f^n by 

Prob(£^(a)<n) = ^. 

Then Dn(t) is a generating function for the /iVn-Q In fact 

Dn{t) = fNnJTTK^ . (1-7) 

Recently, E. Rains [|l^ gave an elegant proof that 

/;Vn = i?n(|tr(f/)P^), (1.8) 



which can be shown to be equivalent to ( |1 . 7|) by a simple argument. Using the relationship 
( |1.7D a sharp asymptotic result for the distribution function of the random variable i'Ar(o") 
was recently obtained by J. Baik, P. Deift and K. Johansson [Q]. And at the same time they 
discovered yet another connection with Painleve. 

Their main result, which was quite difficult, was an asymptotic formula for -D„(t) which 
we now describe. Introduce another parameter s and suppose that n and t are related by 
n = 2t + st^^^. Then as t ^ oo with fixed s one has 

lime-'" D,,^,Mt) = ns)- (1-9) 
Here F is the distribution function defined by 

F{s) = exp(^- J^'^{x-s)q{xfdx^ (1.10) 

where q is the solution of the Painleve II equation 

q" = sq + 2q^ (1.11) 



^Gessel in docs not write down the symbo l of the Toephtz matrix, nor does he mention random 
matrices. But in hght of the well-known formula (L2) and the subsequent work of Odlyzko et al. § and 
Rains ||lo|] , we b elieve it is fair to say that the connection with random matrix theory begins with the 



discovery of (|l.7| ) 
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satisfying q{s) ~ Ai(s) as s — oo. (For a proof that such a solution exists, see, e.g. 0.) 
Using a "de-Poissonization" lemma due to Johansson these asymptotics for the generating 
function Dn(t) led to the asymptotic formula 



hm Prob ( ^^^^^7/0 < ^ ) = Fis) (1.12) 

for the distribution function of the normalized random variable (£Ar(o") — 2^/N)/N^^^. 

It is a remarkable fact that this same distribution function F was first encountered in 
random matrix theory where it gives the limiting distribution for the normalized largest 
eigenvalue in the Gaussian Unitary Ensemble of Hermitian matrices. More precisely, we 



have for this ensemble 11 



lim Prob ((KUN) - V2N) V2N^''^ < s) = F{s). 

N—^oo VV / / 

In connection with these results just described, we shall do two things. We show, first. 



how one might have guessed the asymptotics (|1.12| ). More precisely, we present a simple 
argument that if there is any limit theorem of this type, with F{s) some distribution function 
and with some power A^" replacing N^^^, then necessarily a = 1/6 and F is given by ( 1.10 ) 



with q a solution to ( |1 . 1 ip . (The boundary condition on q, however, cannot be anticipated.) 
This conclusion is arrived at by considering the implications of ( |1.9|) with t^^^ replaced by 
t^" for the recurrence formula ( |l.(j| ). 

Secondly, we derive analogues of ( |1.8D and ( |1.7D for the subgroup On of "odd" permuta- 
tions of iSat .0 These are described as follows: if N = 2k think of iSat as acting on the integers 
from —k to k excluding 0, and if N = 2k + 1 think of Sjy as acting on the integers from —k 
to k including 0. In both cases a G is called odd if cr{—n) = —cr{n) for all n. The number 
of elements in the subgroup On of odd permutations equals 2^^ k\ in both cases. Therefore 
if bNn equals the number of permutations in On having no increasing subsequence of length 
greater than n, 

FToh{iN{a) < ^) = Iff, (1-13) 



where the uniform measure is placed on Ctv- Rains [|TU| proved identities analogous to ( |1.8| ) 
for these probabilities. Using these we are able to find representations for the two generating 
functions 

Gn{t) = E ^SfcnTrjT^, Hnit) = ^ WnTTyT^, (1-14) 
fc>0 v'^v fc>0 v'^v 

analogous to the representation (|1.7| ). (See Theorem 1 below.) The same determinants Dn{t) 
arise as before but in the representation for Hn{t), whose derivation uses the machinery 
developed in earlier sections, the quantities Un also appear. Once the representations are 
estabhshed we can use ( |1.9| ) and Johansson's lemma to deduce the asymptotics of ( |1.13| ). 



•^Our terminology for On is not standard. For N — 2k one usually denotes On by Bk, the hyperoctaliedral 
group of order 2*^/;;! which is the centralizer of the reversal permutation in Sn ■ Elements of Bk are commonly 
called signed permutations. Similar remarks hold for iV = 2fc + 1. 
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Distr 


a S K 


F 
Fa 


-1.77109 0.9018 0.224 0.093 
-1.26332 0.7789 0.329 0.225 



Table 1: The mean (/i), standard deviation (a), skewness (S) and kurtosis (K) of F and 
Fo '■= F"^. 




-5 -4 -3 -2-1 12 



Figure 1: The probabihty densities / = dF/ds and fo = dFo/ds. 



We show that as ^ oo we have for fixed s 



where F{s) is as in ( |1.12| ) 



In Table 1 we give some statistics of the distribution functions F and Fo := F'^. In 
Figure 1 we graph their densities. 

II. The integral representation for 

We write 

A = T„(z-i), A' = T„(z). 
Thus A is the backward shift and A' is the forward shift. It is easy to see that 

T„(z-i /) = T„(/) A + /+ ® 5+ = AT„(/) + 5-® /-, (2.1) 

T„(z /) = r„(/) A' + /- ® r = A' T„(/) + 5+® /+, (2.2) 

where 5^ and were defined above and a®b denotes the matrix with j,k entry ajbk- 
Relation (p.l|) holds for any / but (p.2|) uses the fact that = /fc. 
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To derive ( |1.4| ) we temporarily reintroduce variables r and s and set f{z) = e^^^^''^ ^ 
so D„ and T„ are functions of r and s. Of course we areF interested in D„(t,t). We shall 
compute 

dl, los. DJr, s] 



r=s=t 



in two different ways. 
Using the fact that 

ds logD^{r,s) = tr T^{f)-' T^{dJ) = tr TMT' Tn{z-' f) , 

then differentiating with respect to r, we find that 

dl log D„(r, s) = tr [r„(/)-i T„(/) - T„(/)-i /) T„(/)-^ r„(^-^ /)] 

= tr [/ - T„(/)-i T„(z /) T„(/)-i T„(z-i /)]. 

We now set r = s = t. Since / is now as it was, we can use and (|2.2|) . If we multiply 
their first equalities on the left by T„(/)~^ and use the notation = Tn{f)~^f^ we obtain 

TnifV Tn{z-' /) = A + ® 6+, T„(/)"i r„(^ /) = A' + ® r . 

Hence the last trace equals that of 

I -{A' + u~ S~){A + (g) 5+) 

= / - A' A - (g) A'5" - A'm+ (g) (5+ - (5", M+) (g) 5+. 
The trace of / — A' A equals 1, and 

A'6' = 0, tr A'u+ ® 5+ = (A'm+, 5+) = {u+ , A6+) = 0, 

so we have 



d log Dn{r, s] 



r=s=t 



i-(r, 



But (n-, 5+) = (5-, u+) = (5-, T„(/)-7+) = ^n- Therefore 



log D„(r,s) 



r=s=t 



1 - t/ = $ 



(2.3) 



Now let us go back to general r and s. For any p > Cauchy's theorem tells us that the 
j, k entry of equals 



27ri J\z\ 



2m 



2 =1 



It follows that Dnir, s) = Dn{pr, s/p), and by analytic continuation this holds for any 
(complex) p. Setting p = Js/r we see that 
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It follows that Dn{r, s) is a function of the product rs, as stated in the Introduction, and 
that ^ 

Adl, log Dn{r, s)l^^^^ = ^\og D^{t,t) + 1 A iogD„(t, t). 

Comparing this with (|2.3| ) we see that we have shown 

(P Id 
— logD„(t,t) + --logD„(t,t) =4$,. 



This gives the representation ( [1.4|) . 

Of course it remains to show that this satisfies (|1.3| ). We do this by first finding a 
formula for dUn/dt and then finding relations among the various quantities which occur for 
different values of n. 



III. Differentiation 



In addition to = T„(/) ^f"^, we introduce = T„(/) ^6"^ and we compute some 
derivatives respect to t. First, 

jT^f) = T4iz + z-')f) = T4f) (A + A') + /+ ® 5+ + f-®5- 

by the first equalities of ( PTT| ) and (|2.2D, so 



d 



dt 
Next, 



r„(/)-i = -T„(/) 



-1 dTr^if) 
dt 



( fo \ 

fi 

fn-2 
V fn-1 J 



Tnifr' = -(A + A') TnUT' ®v+ -u- ®v-. (3.1) 



h 

fn 

\ fn+1 J 



r„(/)5+ + A/+ + /„,+ir. 



Hence 



df+ 



Multiplying the second equality of (^]T|) left and right by T!„(/) ^ gives 

Tnify' A = A r„(/)-l + - t;- 

Therefore 

T„(/)-iA/+ = Aw+ + u+ /+) - V /+), 
and substituting this into ( |3.2| ) gives 



T„(/)- 



(3.2) 



(3.3) 
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Adding this to ( |3ri| ) applied to gives 



dt 



S+ - A'u+ + V- - [u-, /+)) - u- [v-, /+). 



Taking inner products with 6 in the last displayed formula we obtain (recall the definition 

ofUn) 

^ = -(A'n+, r) + {v^, 5+) (/„+! - («-, /+)) - («+, 5+) (.;+, /^). 

We used the fact, which follows from the symmetry of T„(/), that all our inner products 
whose entries have signs as superscripts are unchanged if both signs are reversed. 

To find (A'm"*", which is the same as (Am~, 6^), we observe that 



A/" 



/ fn \ 

fn-1 

h 

V ; 



Tn{f) S--foS-. 



Applying (|3.3| ) to / therefore gives 

6--fov- = Au- + u+ {v+, n - V- {u-, /-). 

Hence 

-(A'w+, r) = -{Au-, 5+) = («+, 5+) iv+, n + {v-, 5+) (/o - (w+, /+)). 

Thus we have established the differentiation formula 

dUn 



dt 



{v-, 5+) (/o - {u\ n) + {v\ 5+) - (n-, /+)) . 



(3.4) 



IV. Relations 



New quantities appearing in the differentiation formula are 

Vn^ = (^^^ 5+). 

There are others but we shall see that they may be expressed in terms of these (with different 
values of n), as indeed so will Un- We obtain our relations through several applications of 
the following formula for the inverse of a 2 x 2 block matrix: 







"A 


c 


d) 





(A-BD-^C)-^ X 



(4.1) 



X 



Here we assume A and D are square and the various inverses exist. Only one block of the 
inverse is displayed and the formula shows that A — BD^^C equals the inverse of this block 
of the inverse matrix. At first all that will be used about / is that T„(/) is symmetric. 
(There are modifications which hold in general.) 

We apply ( |4.1|) first to the (n + 1) x (n + 1) matrix 



/ 

/i 





/o 



1 \ 



V fn fn—1 



fn- 



fo ) 



with A = (0), D = T^if), 5 = (0 ■ ■■ 1), C = /+. In this case A - BD-^C = 
— (Tn{f )^^ 6~) = —Un- This equals the reciprocal of the upper-left entry of the inverse 
matrix, which in turn equals (—1)"' times the lower-left n x n subdeterminant divided by 
Dn. Replacing the first row by (/o /i ■ ■ ■ /«) gives the matrix 



/ fo 
fi 



fi 
fo 



V fn fn—1 



fn \ 
fn-1 

fo J 



Tn+l{f). 



The upper-right entry of its inverse equals on the one hand and on the other hand (— 1) 
times the same subdeterminant as arose above divided by -D„+i- This gives the identity 



Dn 



+1 



n+l 



n+1 



Dn 



(4.2) 



n+l 



(If we consider the polynomials on the circle which are orthonormal with respect to the 
weight function / then the right side above is equal to the constant term divided by the 
highest coefficient in the polynomial of degree n. Therefore our — f/„ equals the Sn-i of 0].) 

If we now take A to be the upper-left corner of Tn+i{f) and D the complementary Tn{f) 
then C = f^ and B is its transpose, and we deduce that 



To evaluate fn+i 



fo - {u\ /+) 



(4.3) 



n+l 



/^), the other odd coefficient appearing in 



we consider 



/ fo 

fi 

fn 
\ fn+1 



fl 

fo 

fn-1 
fn 



fn 
fn-1 

fo 

fl 



fn+1 
fn 



\ 



fl 

fo j 



Tn+2{f). 



We apply to this an obvious modification of (|4.1| ), where A is the 2x2 matrix consisting of 
the four corners of the large matrix, D is the central Tn{f), C consists of the two columns 
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and / and B consists of the rows which are their transposes. Then 
A-BD-^C = 

[ - (w-, /+) /o - /+) 
and our formula tells us that this is the inverse of 



This gives the two formulas 



n+2 



'^n+2 '^n+2 



fn+l -{U , 



-v: 



n+2 



Ki+2 ~ Ki+2 



Comparing the first with ( [4 .31) we see that 



*^n.+2 ^n+2 ~ ^n.+l *^n+2' 



and therefore that the preceding relations can be written 



/o - {u\ n - 

Notice that (g^) and (^) give 



fn+l - {U , /+) 



1 K 



n+2 



Ki+1 Ki+2 



2 



This is our $„. 



The relations we obtained so far in this section are completely general. The 
(|1.6D, however, depends on our specific function /. Integration by parts gives 

2m J 

Hence if M = diag (12 ■ ■ ■ n) we have 

MT„(/) - T„(/) M = t T„((z - z-') /), 
and by the first identities of ( p^.l] ) and ( |2.2| ) this equals 

Tnif) (A'-A) + r ®r-/+®(5^ 



so 



Tnif)-' M - MTnif)-' = t [{A' - A) T„(/)-i + u- y- - u+ ® 
Applying this to 6~ gives 



nv — M V = t 



(A' -A)v- + u~ {y-, 5-) - M+ r ) 
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Now say 



( /O - /2 \ 

/i — /s 



Mf+ = t _ 

fn-2 ~ fn 
\ fn-l ~ fn+1 I 

whereas (this is relevant since the transpose of A' — A is A — A') 

/ h \ 

/s — /i 



(A - A') r 



fn fn—2 



Therefore 



M /+ + t (A - A') /+ = t (/o 5+ - /„,+! r ), 
and so taking inner products with in ( [4.8| ) gives 

n 



t 



(v-, n = h (v-, 5+) - {v-, r ) + /+) (v-, r) - /+) iv+, r ) 



or equivalently, since {v , /+) = f/„, 

f/n = (/O - /+)) - (/n+1 - {n-, n) K 

Using (|4.5| ) we rewrite this as 
n 



n 
1 



t 



Un 



Using this, {WM and P^) we arrive at (KM). 



V. Painleve V and Painleve III 



We first show that satisfies ( |1.3D . Our formula ( |3.4| ) for dUn/dt can now be written 



71+1 



•^71 _ ( ^ n+2\ _ /i tt2\ (tt tt \ 

^n+2 ^n+2 



by ( [4 .51 ), ( ^I6| ) and ([4.2|). Adding and subtracting ( |1.6| ) gives us the two formulas 



^ = yf/n + 2t/.+l(l-t/„^), 



dt 



n 



-Un-2Un-l{l-Ui 



(5.1) 



(5.2) 
(5.3) 
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These are equations (4.5) and (4.6) of [Q. As was done there, we solve ( p.2| ) for Un+i in 
terms of f/„ and dUn/dt and substitute this into (|5.3| ) with n replaced by n + 1. We get a 
second-order differential equation for f/„ which is equivalent to equation (|1.3|) for $„ = l — U^- 



Next we show that Wn = Un/Un-i satisfies ( |1.5| ). In computing the derivative of Wn we 
use (|5.3| ) to compute the derivative of f/„ and (|57^) with n replaced by n — 1 to compute the 
derivative of f/„_i. We get 

7n — 1 

< = W^-2 + AUl-2 Wl (5.4) 



Using (|5.3| ) once again we compute 



{Ul)' = 2U^ (->„-2f/„_i(l-fZ„2)U-2>„^-4.^"^^ 



Differentiating ( ^.4|) and using this expression for (f/^)' we obtain a formula for W^^ in terms 
of Wn, and t/^. Then we solve ( ^^ ) for f/^ in terms of VF„ and W^. Substituting this 
into the formula for W" gives (|1.5|). 



In order to specify the solutions of the equations (|1.3|) and (|1.5|) we must determine the 
initial conditions at t = 0. Clearly $^(0) = 1 but this does not determine uniquely. One 
can see that ^i'^HO) = for A; < 2n and that what determines $„ uniquely is $^^'^•'(0). We 
shall show that 

<^"H0) = -^^. (5.5) 

By ( [4.2| ) Un = Ki+i/Ki+i- Now V^+i is the upper-left corner of T„+i(/)^^ and so tends 
to 1 as t — 0. So let us see how V^+i, which is the upper-right corner of T„+i(/)~^, behaves. 
More exactly, let us find the term in its expansion with the lowest power of t. 

We have 

k ^■ 

= E Y,C{k,j)e-^^'^-'^^' = j:^^Cik,U + k)/2)e~^^'. 

0<j<k \j\<k ^■ 

This gives 

TnU) =11 tr, Cik, (j + k)/2) a:'- = J + ^ ^ C{k, (j + k)/2) A^. (5.6) 

\j\<k 

(Here A-' denotes the usual power when j > 0, but when j < it denotes A' '"''.) 
We use the Neumann expansion 

Tnif)-' = I + E(-l)' E S C{k, (j + k)/2) 

1>1 \ k>0 

\\j\<k 
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If we expand this out we get a sum of terms of the form coefficient times 

Now the product A-'^ ■ ■ ■ A-^' can only have a nonzero upper-right entry when ji + ■ ■ ■ +ji > n. 
Since each < k the power of t must be at least n, and this power occurs only when each 
ji = k. That means that we get the same lowest power of t term for the upper-right entry if 
in ( |5.6|) we only take the terms with j = k, in other words of we replace T„+i(/) by 

fc>0 

The inverse of this operator is e~*^ and the upper- right corner of this matrix is exactly 
This shows that 

and so 

$n = l-f/„^=l-^ + 0(t-+^), 

which gives ( [5. 51 ). We also see that Wn = Un/Un~i satisfies the initial condition 

Wn{t) = -- + 0{f). 

n 

Using the differential equation (|1.5|) together with this initial condition we find 
W it) = -'-- '-^ + 6)f 



VI. Painleve II 



We present a heuristic argument that if there is any limit theorem of the type (|1.12| ), 
with some distribution function F(s) and some power A^"" replacing A^^^^, then necessarily 
a = 1/6 and F is given by ( |1.10D . First we note that Johansson's lemma (which we shall 
state in the next section) leads from ( 1.12 ) to (]_J) with the power t^^^ replaced by t^". We 



assume that F is smooth and that the limit in ( |1.9|) commutes with d/ds, so that taking the 
second logarithmic derivative gives 



lim -r^log D2t+st2^ = 

where is now defined by —q^ = (log F)" and q is defined to be the positive square root of 
(for large s). 
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Since changing n = 2t + s t^" by 1 is the same as changing s by t , we have the large t 
asymptotics 

log + log D„_i - 2 log Dn ~ t-'" ^ log Z^2t+st2" ~ -f '° 

On the other hand = Dn-i/Dn and so 

^"+^ ^"-^ = = 1-U^ (Ql) 

by (|;6D. We deduce 

log(l-t/„2)~-r^"g(.)^ (6.2) 



Now the Un are of variable sign, as is clear from ( |1.6|) . Let us consider those n going to 
infinity such that 

Un-1>0, f/„<0, f/n+l>0, (6.3) 

and write ( |1.6| ) as 

t {Un+i + Un-i + 2 f/„) (1 - Ul) = -{n - 2t) Un - 2t Ul (6.4) 

Because of ( |6.3[ ), ( p.2| ) and the fact that changing = 2t + s by 1 is the same as changing 
s by we have when t is large, 

?7„+i + t/„_i + 2f/„~t-6°g"(s). 

Since also 

n-2t~st^", [/„ ~ -t-2°g(s), 
( |6.4|) becomes the approximation 



Let us show that a = l/6. IfQ;>l/6 then letting t ^ oo in (|6.5| ) gives q{s) = and so F 
is the exponential of a linear function and therefore not a distribution function. If a < 1/6 
then dividing by t^"^" and letting t ^ oo in ( |6.5D gives q"{s) = 2q^. Solving this gives two 
sets of solutions 

fio dq 
s = ± , ^ + C2, 

where go, ci and C2 can be arbitrary. Now q is small when s is large and positive, so s is 
large and positive when q is small. Therefore we have to have the + sign and we must have 
Cl = 0. Then -F(s) is of the form \s — c\~^ times the exponential of a linear function and 
therefore is not a distribution function. The only remaining case is a = 1/6, and then ( |6.5| ) 
becomes ( |1 . 1 1| ) . It follows that -F(s) must be given by ( l.l(j| ) times the exponential of a 



linear function. This extra factor must be 1 since ( |1.1CI| ) is already a distribution function 



Now to derive this we assumed that the n under consideration were such that (|6.3| ) held. 
We would have reached the same conclusion if all the inequalities were reversed. If n — >■ cxd 
in such a way that, say, t/„_i and ?7„ have one sign and ?7„+i the other, then (the reader can 
check this) we would have reached the conclusion g = 0. Thus the only possibility for q to 
give a distribution function occurs when a = 1/6 and q satisfies ( |1.11| ). 
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VII. Odd permutations 



Recall that bNn equals the number of permutations in On having no increasing sub- 



sequence of length greater than n. The representations of Rains [10| for these quantities 
are 



hkn = E^{\tT{Uy\^), (7.1) 
hk+ln = Ej\tT{U'ftT{U)\'). (7.2) 



Theorem 1. Let Gn{t) and Hn{t) be the generating functions defined in ( |1.14|) . Then 



Gn{t) = (7.3) 





n 


even 


Dn^{t)Dn±l{t), 


n 


odd, 




n 


even 


Dn-l(t)Dn + l (t), 
2 2 


n 


odd. 



Hn{t) = (7.4) 



We prove a lemma which gives a preliminary representation for the generating function 
in terms of other Toeplitz determinants. Let 



and define 



Lemma. We have 



Dn{ti,t2) = Dn = detTn{g). 



G,,it) = D^{0,t) (7.5) 
19^ Id^ 



^n(t) = 7^(0, t) + 7^(0, -t). (7.6) 



The proof of ( [73| ) is essentially the same as the proof that ( |1.7| ) and (|T]B|) are equivalent. 
First observe that 

((tr(t/2) + 1^)''^) = E {^Av'T^Wf"'"^) ■ 

Each summand with m ^ k vanishes since by the invariance of Haar measure replacing each 
U by C,U , with C, a complex number of absolute value 1, does not change the summand but 
at the same time multiplies it by (^4m-4A: 'jj-^^g^ 
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Hence ( |7.1| ) is equivalent to 
Therefore if the eigenvalues of U are e*^^ , ■ ■ ■ , e*^" we have 

The last step follows from ( p..2|) . 

This gives ( [7.5[) . To prove ( [7.6|) we use (|7.2| ): 



62fc+in = (^tr(t/)tr(t/2)Hr(f/)tr(f/2) 

= ^I^K ((tr(f/) +Mf7))' (tr(f/^) 
by expanding the right side as before. Hence 

Hn{t) = IE-^E^ ((tr(f/) +te(Z7))' (tr(f/^) +W^"'' 
= 2En([Yl COS 9j) ^ cosh (2t ^ cos 9j 



En (E cos^.y n e"-^^^^ + K (E cos^^.y n ^ 



-2t cos 20, 



From the last equality (|7.6|) follows. 

To prove the theorem we consider (|7.3| ) first. Observe that Dn{0,t) is the determinant of 
Tn{h) where h{z) = f{z'^). It has Fourier coefficients 

h2k = fk, h2k+l = 0. 

Let us rearrange the basis vectors Cq, ei, • • • , e„_i of our underlying n-dimensional space as 

Co, 62, ■ ■ ■ , ei, 63, ■ ■ ■ . (7.7) 

Then we see from the above that Tn{h) becomes the direct sum of two Toeplitz matrices 
associated with /, the orders of these matrices being the sizes of the two groups of basis 
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vectors in ( [7.7| ). If n is even both groups have size n/2 whereas if n is odd the sizes are 
{n ± l)/2. Since £)„(0,)f:) = detT„(/i) is the product of the corresponding Toeplitz determi- 
nants associated with /, we have (|7.3|). 

The proof of ( |7.4| ) is not so simple. We have 



1 d^R 



bn dtl 

Now, as in the computation leading to ( p.3|) , 

dl log bn{h, h) = tr {Tnigy' Tnidlg) - Tn{g)-' T^idt.g) T^ig)-' T^idt.g 



tr 



T4g)-' Tniiz + z-^fg) - T^g)-' T4{z + z~')g) T^g)-' T^{{z + z-')g) 



This is to be evaluated first at ti = 0, t2 = i- Since (7(2;, 0,t) = /(-z^) = h{z) we must 
compute 



tr 



T„(/i)-i T„((^ + z'^fh) - Tnih)-' Tniiz + z~')h) Tnihy' T„((^ + z-')h) 



We write = Tn{h) ^h^, so that the are associated with h just as are associated 
with /. Using and Q we find that 

Tn{hy^ Tn{{z + z-^)h) Tnihy' Tn{{z + z-^)h) = {A' + ® 6' + K + ® 

and from this that 

tr (t„(/i)-i Tn{{z + T„(/i)-i T„((^ + z-^)h)) = 

2n-2 + 2{w-,k5') + 2(w+, A'(5+) + 2(r , (w", 5+) + (5" , w-)^ + 
Consequently 

tr [t„(/i)-i T„((z + - T„(/i)-i Tn{{z + T„(/i)-i T„((z + z-')h) 

= tr (T„(/.)"iT„((^2^z-2)/i)) 

+2 - 2{w-, M-) - 2(w+, A'(5+) - 2(r , w+) (u;-, 5+) - (r , ti;-)^ - 
This is (9^^^ log -D„(0, if:). Similarly, we find 

dt, log^„(0,t) = (w;+,(5+) + (w;-,r), 

so that when ti = 0, t2 = t 
1 S^A 



A /),2 -5' log^„+(5t,logD„) =tr(T„(/i)-iT„((^2 + z-2)/i 
+2 - 2(w;-, Ar ) - 2{w+, A'6+) - 2(w+, r ) (w", 5+) + 2(w+, 5+) (w", r ). 
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Since Tn{h) is symmetric all superscripts in the symbols in the inner product may be reversed 
as long as we interchange A and A'. We therefore have shown 



tr (r^h)-' Tniiz^ + z~^)h))+2-A {w^, A'6+)-2 {w+, r )'+2 {w+, 6+^. 



Let us rearrange our basis elements as in ( [7.7| ) and suppose the first group has rii vectors 
and the second group has n2- Then Tn{h)~^ becomes the matrix direct sum 

T^M)-' 

and Tn{h)~^Tn {{z^ + z~'^)h) becomes 

T„,(/)-iT„,((^ + z-i)/) 

T^,{f)-^T^,{{z + z-^)f) 

By a now familiar computation we find from this that 

tr [tM~^ Uiz' + z-')h)) = 2 «, 5+ ) + 2 «, 5+ ), 

where and 6:^ denote the quantities m"*" and associated with the index m. We use 
similar notation below. To continue, after rearranging our basis we have the replacements 



and 



/ 

V fn2 

6- ^ 







w 



A'6 



if n is even, 



/ 



if n is odd. 



6, 



n2 



It follows from these that 



and that 6^) = (m^j, if n is even and (5^, w^) = if n is odd. 
If we modify our notation by writing 

SO that U~ is what we have been denoting by Un, the above gives 

1 - 



n even. 



l + f/i+i(t) -f/li(t), nodd. 



To evaluate our quantities at (0, —t) we observe that if C is the diagonal matrix with 
diagonal entries 1, —1, 1, ■ ■ ■ , (—1)" and we replace t by —t then we have the replacements 
Tn{f) CTn{f)C and f'^ —Cf'^ and therefore — > —Cu~^. Therefore also = 
{u~^, 6^) —Un and f/^ = (m^, 5^) (— l)""*""^ f/~. Hence in the last displayed formula 
U~i^ is an even function of t whereas U^n±i)/2 ^'^^ functions of t. Also, DniS^^t) is an 
even function of t. Thus 



Hn{t) = Gn{t) X 



1 — f/s. (t)^, n even, 
1, n odd. 



Recalling ( |4.6| ) and the general fact V^{t) = Dm-i(t) / Dmit) we obtain ([7.4[). 

Theorem 2. Let £Ar(cr) denote the length of the longest increasing subsequence of a in the 
subgroup Oj\f of S]\f . Then 

where F{s) is as in ( |1.12|) . 

For the proof we shall apply Johansson's lemma P], which we now state: 

Lemma. Let {Pk{'n)}k>o be a family of distribution functions defined on the nonnegative 
integers n and V5„(A) the generating function 

(Set Po{n) = 1.) Suppose that for all n, k > 1, 

Pk+i{n) < Pk{n). (7.8) 
If we define Hk = k + Ay/k \ogk and Uk = k — 4y/k\ogk, then there is a constant C such that 



C C 
p < Pk{n) < M^k) + -p 



for all sufficiently large k, < n < k. 



This allows one to deduce that Pk{n) ~ ^Pn{k) a.s n, k ^ oo under suitable conditions, 
and is how one obtains the equivalence of ( |1.9| ) and ( |1.12| ). We shall apply the lemma to the 
distributions functions 

^:(A) = e-'G„, (^) = e-' ^ F^kin)^ , 

^ ^ k>Q 
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where 



Fj^{n) = Prob(^jv(o^) < n) 



ONn 



(7.9) 



when N = 2k oi 2k + 1. To apply this lemma we must prove another 

Lemma. We have -F/v+2 < F^q for all A^. 

We show this simultaneously for N = 2k and N = 2k + 1. Take a a G On+2 and remove 
the two-point set {—1, 1} from its domain. Then a maps the remaining A^-point set one-one 
onto another A^-point set. If we identify both of these sets with the integers from —k to 
k (including or excluding depending on the parity of N) by order-preserving maps, then 
under this identification the restriction of cr becomes an element of 5jv . In fact it becomes 
an odd permutation because the two identification maps are odd. Thus we have described a 
mapping a ^(c) from to Ojy. The mapping is 2fc -|- 2 to 1 and is clearly onto. It is 

also clear that ^Ar(J^(cT)) < £^+2(0'), from which it follows that bN+2 n < (2/c + 2)bNn for all 
n. The assertion of the lemma follows upon using ( [7.9|) . 

Theorem 1 tells us that 



e^*>:(2t^) 







n 


even, 


Dn-l(t)Dn+l 
2 2 




n 


odd. 


(l-f/fW^) 




n 


even 


Dn-l(t)Dn+l 
2 2 




n 


odd. 



From the asymptotics ( |1.9| ) we find that if we set 

2t + st^'^ 



n 
2 



then 



Setting t = y2k/2 gives 



hm ^l{2e) = F{sf. 



hm vl{k) = Fisf. 

k—rOO 

Johansson's lemma tell us that when runs through the even integers 2fc, 
lim Fn{2^ +2'^'^sN'^'^) = lim F/v(n) = F(s)2. 



For A^ running through the odd integers we obtain the same relation. For this we have to 
use the additional fact that Un/2{t) — as t — 00. If we recall (|6.1|) we see that this follows 
from the asymptotics of -D„/2- Thus the proof is complete. 
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